Systemd 启动缓慢之谜：一次排查经历

文章背景与核心概要

本文探讨了一台 Ubuntu 26.04 服务器上出现的 SSH 登录延迟问题，该问题导致用户登录时出现超过 3 秒的卡顿。通过使用修改版的 extrace 工具和 systemd-analyze，作者将性能瓶颈定位在用户级 systemd 会话管理器的单元加载阶段。

调查显示，该问题并非 systemd 本身的 Bug，而是系统底层在处理单元枚举时出现了异常延迟。本文详细记录了排查过程中的技术挑战、所使用的调试工具，以及针对此类动态用户空间进程进行诊断的复杂性，为处理类似系统性能异常提供了参考思路。

1. 症状描述

正如最初在 Fediverse 上分享的那样：

今天的谜题：在唯一的一台系统上（使用共享 NFS 主目录），登录需要 3 秒以上才能启动，因为 /usr/lib/systemd/systemd --user 在运行 systemd-xdg-autostart-generator 和执行后续极其晦涩的登录过程之间停滞不前。

我甚至不知道从哪里开始，我能查到这一步全靠一个修改版的优秀工具 extrace，它能报告进程启动的时间戳。

(注：尽管最初的帖子中未提及，但这些都是标准的 SSH 登录。)

Today's mystery: on exactly one system (with shared NFS home directories), logins take 3+ seconds to start because /usr/lib/systemd/systemd --user sits on its hands between running systemd-xdg-autostart-generator and running the next step of a remarkably opaque login process.

I don't even know where to start and I only got this far with a hacked version of the great extrace that also reports a process start time number.

(Note: Although omitted in the initial post, these are standard SSH logins.)

这种行为仅限于一台 Ubuntu 26.04 系统。其他共享完全相同的软件配置和 NFS 挂载主目录的机器并没有出现这种延迟。此外： * 这种缓慢现象在多次重启和修改主机名后依然存在。 * 一旦初始的 systemd 用户会话建立，后续的 SSH 登录将瞬间完成。 * 该问题影响多个用户账户，而不仅仅是单个用户。

This behavior is isolated to a single Ubuntu 26.04 system. Other machines sharing the exact same software configuration and NFS-mounted home directories do not exhibit this delay. Furthermore: * The slowness has persisted across multiple reboots and a hostname change. * Once an initial systemd user session is established, subsequent SSH logins are instantaneous. * The issue affects multiple user accounts, not just a single user.

变通方案

对于我自己的账户，我使用 SSH 主连接多路复用（multiplexing）的变通方法缓解了这个问题：

ssh -o "ControlPersist 30d" -N -M <host> -f

这使得会话无限期保持活跃，从而消除了重复初始化新 SSH 连接和用户会话的需要。

The Workaround

For my own account, I mitigated the issue using an SSH master connection multiplexing workaround:
ssh -o "ControlPersist 30d" -N -M <host> -f
This keeps the session active indefinitely and eliminates the need to initialize a fresh SSH connection and user session repeatedly.

2. 使用 `systemd-analyze` 定位瓶颈

要检查会话启动性能，可以运行 systemd-analyze --user dump。运行此诊断程序揭示了一个明显的问题：

Timestamp units-load-start: Thu 2026-07-09 18:40:30 EDT
Timestamp units-load-finish: Thu 2026-07-09 18:40:33 EDT

仅仅为了加载单元就延迟了三秒——这发生在 systemd 实际开始执行任务之前，但在单元生成器运行之后——这是极不正常的。

To inspect session startup performance, you can run systemd-analyze --user dump. Running this diagnostic revealed a clear problem:
Timestamp units-load-start: Thu 2026-07-09 18:40:30 EDT
Timestamp units-load-finish: Thu 2026-07-09 18:40:33 EDT
A three-second delay just to load units—occurring before systemd actually starts executing tasks, but immediately after unit generators have run—is highly abnormal.

3. 代码调查

进一步追踪直接指向了 systemd 的管理器源代码 (src/core/manager.c)。延迟发生在两个主要调用附近： 1. manager_enumerate_perpetual() 2. manager_enumerate()

（也在 manager.c 第 1820 行中引用）。

这些函数枚举并设置已知的 systemd 单元。用户会话通常只包含少数几个永久单元（如 -.slice 和 init.scope）。非永久枚举处理设备、挂载点和交换空间，随后触发单元加载队列。

Tracing this further led directly to systemd's manager source code (src/core/manager.c). The delay occurs around two primary calls: 1. manager_enumerate_perpetual() 2. manager_enumerate()

(Also referenced in manager.c line 1820).

These functions enumerate and set up known systemd units. User sessions typically contain only a few perpetual units (such as -.slice and init.scope). Non-perpetual enumeration handles devices, mounts, and swaps, subsequently triggering the unit load queue.

4. 调试挑战

文件系统活动： 使用 opensnoop-bpfcc 监控打开的文件没有显示出明显的错误，尽管在文件打开活动中确实存在明显的间隙。
系统调用追踪： 虽然更深入的 strace 日志可能会有所帮助，但在 PID 已知之前，通过 UID 和命令名称来隔离动态生成的用户级 systemd 进程存在巨大的追踪挑战。

File System Activity: Using opensnoop-bpfcc to monitor opened files showed nothing blatantly wrong, though notable gaps in file-opening activity were present.

System Call Tracing: While deeper strace logs could help, isolating a dynamically spawned user-level systemd process by UID and command name before its PID is known presents a significant tracking challenge.

5. 潜在的根本原因

怀疑点并不在于 systemd 本身。相反，systemd 似乎正在查询或与某个在此特定主机上异常缓慢的子系统进行交互。

这台机器上还存在其他细微的性能异常——例如 dpkg -S 有时会变慢——但我们的指标监控系统显示没有明显的硬件瓶颈（如 CPU 节流或内存耗尽）。对于最终可能归结为简单的局部配置或硬件怪癖的问题，很容易陷入过度复杂化诊断的陷阱。

Suspicions point away from systemd itself. Instead, systemd appears to be querying or interacting with a subsystem that is unusually sluggish on this specific host.

Other minor performance anomalies exist on this machine—such as occasional slowness with dpkg -S—yet our metrics monitoring system shows no obvious hardware bottlenecks (such as throttled CPUs or exhausted RAM). It is easy to fall into the trap of overcomplicating diagnostics for what may ultimately boil down to a simple, localized configuration or hardware quirk.